Data fusion in image systems refers to a technique which generates a single image by mathematically fusing respective pieces of image information photographed by multiple sensors, and is a technology which is capable of overcoming the limitation of image information provided by a single sensor. The development of data fusion began in earnest from the early 1990s with the active use of earth observation satellites. It has been explored by the research institutes of various countries, and has resulted in the development of technology optimized for Korea Multi-purpose Satellites (KOMPSAT)-2 and -3 and research into utilization in the field of earth science.
Generally, optical satellites provide both a high-resolution panchromatic image and a low-resolution multispectral image. A panchromatic image is advantageous to object extraction and reading due to the provision of high-resolution image information, whereas a multispectral image is advantageous to land coverage classification through the analysis of the spectral characteristics of objects due to its high spectral resolution. Although it is effective to provide a high-resolution multispectral image in order to achieve ideal use, spatial resolution and spectral resolution are in a trade-off relationship, and thus it is physically difficult to provide a high-resolution multispectral image. Accordingly, an early data fusion technique was developed with the goal of generating a multispectral image having resolution forcibly increased by mathematically fusing a high-resolution panchromatic image and a low-resolution multispectral image photographed by the same optical satellite. This has achieved technical outcome via Korean Patent No. 10-1132272 entitled “Method of Generating Fused High-resolution Satellite Image by Using Fusion Coefficient Reflecting Spectral and Spatial Characteristics of Images,” Korean Patent No. 10-1291219 entitled “Method and Apparatus for Fusing Panchromatic Image and Multispectral Image,” etc.
However, a panchromatic image and a multispectral image provided by the optical satellite are photographed in a way similar to the way that a human views real world geographic features. Accordingly, they are advantageous to visibility and readability, but have limitations in terms of information about land surface characteristics, such as the temperature, roughness, water content, etc. of a land surface. The same is true of a fused image generated by fusing the above two images. Accordingly, recently, research has been conducted into the development of a data fusion technique capable of providing higher-level information by fusing pieces of information provided by different satellites.
In particular, the image information provided by an infrared image is acquired by imaging earth radiation energy, not solar radiation energy. An infrared image can be photographed at night. Since earth radiation energy can be converted into temperature, an infrared image is advantageous to metal/nonmetal detection, and thus can be widely used for military and civil fields, such as the field of object recognition including image classification, target detection, vegetation monitoring, soil moisture content extraction, etc. Earth radiation energy in the wavelength band from 8 to 15 μm is insignificant compared with solar radiation energy. According to the Stefan-Boltzmann law, earth radiation energy is 204 times less than solar radiation energy in the wavelength band from 0.4 to 2.5 μm, and thus it is difficult to acquire a high-resolution infrared image. Therefore, there is an urgent demand for the development of a technique for generating a fused high-resolution infrared image by using a high-resolution panchromatic image.
Meanwhile, Korean Patent No. 10-1051716 discloses a “multi-sensor image fusion method,” including the steps of: acquiring first and second images of the same subject point by means of different photographing sensors; extracting feature points from the first image by means of a FACET-based filter, and extracting feature points from the second image by means of a Harris corner detector; determining a feature point pair, exceeding a threshold value, to be corresponding points by comparing the mutual information between a specific region based on the feature points of the first image and a specific region based on the feature points of the second image based on the feature points of the second image; converting the coordinate unit of the first image into the coordinate unit of the second image, wherein when the coordinate of a converted pixel is not an integer but a real number, bilinear interpolation designed to perform conversion into an integer-type coordinate is applied; and forming a single third image in which the first image and the second image have been registered with each other by placing the feature points of the first image and the feature points of the second image in a corresponding point relationship at the same locations.
This preceding technology is configured to extract feature points of each of a multispectral image and an infrared image and to generate a fused image through corresponding point determination and unit coordinate conversion. Although this preceding technology has the advantage of simultaneously showing the image information of an infrared image and the spectral information of a multispectral image, it has difficulty providing a high-resolution infrared image.
Furthermore, Korean Patent No. 10-1104199 discloses a “visible and infrared image signal fusion apparatus and method,” including the steps of: registering a first input image from the outside based on a preset alignment parameter by means of a first image alignment unit; registering a second input image from the outside based on a preset alignment parameter by means of a second image alignment unit; correcting the gain and offset of the signal of the first input image having passed through the registering process by means of a first gain/offset correction unit; correcting the gain and offset of the signal of the second input image having passed through the registering process by means of a second gain/offset correction unit; generating a Laplacian pyramid for the signal of the first input image, whose gain and offset have been corrected, by means of a first Laplacian pyramid generation unit; generating a Laplacian pyramid for the signal of the second input image, whose gain and offset have been corrected, by means of a second Laplacian pyramid generation unit; receiving the generated Laplacian pyramids from the first and second Laplacian pyramid generation units by a low-band pass filter unit, and performing pyramid step-based low-band filtering; receiving the generated Laplacian pyramids from the first and second Laplacian pyramid generation units by a high-band pass filter unit, and performing pyramid step-based high-band filtering; and generating a fused image by using pyramid step-based Laplacian images filtered via the low-band pass filter unit and the high-band pass filter unit and processing step-based images by means of a fused image generation unit.
This preceding technology is configured to generate Laplacian pyramids, to extract high-frequency information present in a panchromatic image, and to inject the high-frequency information into an infrared image, thereby generating a fused image. Although this preceding technology has the advantage of effectively maintaining infrared image information, it is problematic in that processing speed is low, the boundaries of objects inside an image become ambiguous due to the high-frequency information of a panchromatic image, and an area where the spatial resolution of a fused image is considerably lower than the spatial resolution of a panchromatic image occurs due to a blurring phenomenon.